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5 The invention relates to a motion-compensated image signal interpolation unit 

for generating an interpolated image intermediate a first and a second image, the interpolated 
image being located at a first predetermined temporal distance from the first image and being 
located at a second predetermined temporal distance from the second image, the interpolation 
unit comprising: 

10 motion estimation means for furnishing a first and a second motion vector relating to the first 
and second image; 

furnishing means for furnishing a. first group of samples on basis of values of pixels of the 
first image and the first motion vector and for fiirnishing a second group of samples on basis 
of values of pixels of the second image and the second motion vector, and 

15 filtering means for ordered statistical filtering of the samples of the first and the second group 
to produce a first value of a first pixel of the interpolated image. 

The invention further relates to an image processing apparatus comprising: 
receiving means for receiving an image signal representing a first and a second image; and 
such a motion-compensated image signal interpolation unit coupled to the receiving means, 

20 for generating an inteipolated image intermediate the first and the second image, the 

interpolated image being located at a first predetermined temporal distance from the first 
image and being located at a second predetermined temporal distance from the second image. 

The invention further relates to a method of generating an interpolated image 
intermediate a first and a second image* the interpolated image being located at a first 

25 predetermined temporal distance from the first image and being located at a second 
predetermined temporal distance from the second image, the method comprising: 
furnishing * fi*st and a second motion vector relating to the first and second imago; 
fbnrishing a first group of samples on basis of values of pixels of the first image and the first 
motion vector and for fijrnishing a second group of samples on basis of values of pixels of 

30 the second image and the second motion vector; and 

ordered statistical filtering of the samples of the first and the second group to produce a first 
value of a first pixel of the interpolated image. 

The invention further relates to a computer program product to be loaded by a 
computer arrangement, comprising instructions to generate an interpolated image 
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intermediate a first and a second image, the interpolated image being located at a first 
predetermined temporal distance from the first image and being located at a second 
predetermined temporal distance from the second image, the computer arrangement 
comprising processing means and a memory, the computer program product, after being 

5 loaded, providing said processing means with the capability to cany out: 

furnishing a first and a second motion vector relating to the first and second image; 
fiunishing a first group of samples on basis of values of pixels of fee first image and the first 
motion vector and for furnishing a second group of samples on basis of values of pixels of 
the second image and the second motion vector; and 

10 ordered statistical filtering of the samples of the first and the second group to produce a first 
value of a first pixel of the interpolated image. 



An embodiment of the unit of the kind described in the opening paragraph is 
15 Icnown from the US patent US 5,534,946. In US 5,534,946 a motion-compensated picture 
signal interpolation apparatus is disclosed, comprising switched memories for furnishing 
motion-compensated pixel values corresponding to at least two motion vectors. The motion- 
compensated pixel values are processed by means of an ordered statistical filter. 

When very complex and fest movements in the image cannot be estimated 
20 correctly, for example, because they are outside the estimator range or because the velocity 
field is very inconsistent, estimated motion vectors are incorrect and a motion-compesnsated 
interpolation based on these motion vectors yields erroneous results. Subjectively, artifacts 
arising from such incorrect motion vectors are very disturbing at transients and in the 
neighborhood of stationary objects, since they tend to be concentrated in an isolated, 
25 conspicuous spot on the screen, especially when there is higfr spatial detail. In other "words, 
the quality of the interpolation, and in particular the preservation of edges, depends on the 
quality of the motion vectors. 

It is an object of the invention to provide an image signal interpolation unit of 
30 the kind described in the opening paragraph which is arranged to substantially preserve edges 
relatively independent of the quality of the motion vectors. 

This object of the invention is achieved in that there is a first quotient which is 
substantially equal to a second quotient, the first quotient being determined by a first spatial 
distance between a first one of the samples of the first group and a second one of the samples 
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of the first group and the first predetennined temporal distance, the second quotient being 
determined by a second spatial distance between a first one of the samples of the second 
group and a second one of the samples of the second group and the second predetermined 
temporal distance, 

5 The interpolation, i>e. the computation of pixel values of the inteapolated 

image is based on samples derived from the first image and the second image. However the 
density of samples in the first image is in general different from the density of samples in the 
second image, as the first temporal distance is different from the second temporal distance. 
That means that the spatial distance between the left sample and the right sample of the first 

10 group, Le, the spatial coverage in the first image, is larger or smaller than the spatial distance 
between the left sample and the right sample of the second group, i.e. the spatial coverage in 
the second image. Preferably the number of samples in the first group and the second group 
are mutually equal. The number of samples in the first group and the second group are 
predetermined, i.e. fixed, The spatial coverage in the first image is related to the first 

15 temporal distance and the spatial coverage in the second image is related to the second 

temporal distance. The spatial coverage in the first image is related to the mflvMm error in < 
the first motion vector for which correction is can be achieved. Suppose that the first motion 
vector connects the first pixel of the interpolated image and the central sample of the first 
group of samples. Then the other samples of the first group are located around this central 

20 sample within a spatial environment in which an appropriate sample could be fetched taking 
into account the error in the first motion vector. The spatial environment is related to the first 
temporal distance, since the further away from the first image, the bigger the effective spatial 
deviation being caused by the error in the first motion vector. 

In an embodiment of the motion-compensated image signal interpolation unit 

25 according to the invention the filtering means include a median filter. Preferably a weighted 
median filter. That means that the samples of the first group and of the second group are 
multiplied with respective weighting coefficients and then ordered. As a consequence the 
total set of ordered samples comprises multiple samples which are based on a single sample 
of e.g. the first group of samples, 

30 In an embodiment of the motion-compensated image signal interpolation unit 

according to the invention comprising the weighted median filter, a particular weighting 
coefficient of the weighted median filter for weighting a particular sample of the first group 
of samples is higher than each of the further weighting coefficients for weighting further 
respective samples of the first group of samples, the particular sample being located in the 
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center of the first group of samples, Preferably the particular weighting coefficient is higher 
than a sum of the further weighting coefficients. 

hi an embodiment of the motion-compensated image signal interpolation unit 
according to the invention a second value corresponding to the first one of the samples of the 
5 first group equals a third value of a third one of the pixels of the first image. In other words, 
the first one of the samples directly corresponds to a pixel value of the first image. An 
advantage of this embodiment is that the establishing of the samples is relatively easy. 

In an embodiment of the motion-compensated image signal interpolation unit 
according to the invention a second value corresponding to the first one of the samples of the 
10 first group is computed by means of interpolation of a third valueof a third one of thepixels 
of the first image and a fourth value of a fourth one of the pixels of the first image in a spatial 
environment of the third one of the pixels. In other words, the first one of the samples 
corresponds to a combination of two or more pixel values of the first image, 

A second aspect of the invention provides an image processing apparatus as 
15 defined in claim 9, The image processing apparatus optionally comprises a display device for 
displaying the interpolated image and preferably also the first and second image. The image 
processing apparatus might e.g. be a TV, a set top box, a VCR (Video Cassette Recorder) 
player, a satellite tuner, a DVD (Digital Versatile Disk) player or recorder. 

A third aspect of the invention provides a method as defined in claim 1 1. 
20 A fourth aspect of the invention provides a computer program product as defined in claim 12. 

Modifications of the image signal interpolation unit and variations thereof may 
correspond to modifications and variations thereof of the image processing apparatus, the 
method and the computer program product, being described. 

25 

These and other aspects of the image signal interpolation unit, of the image 
processing apparatus, of the method and of the computer program product, according to the 
invention will become apparent from and will be elucidated with respect to the 
implementations and embodiments described hereinafter and with reference to the 
30 accompanying drawings, wherein; 

Fig, 1 schematically shows two input images and an interpolated image 
intermediate to these input images; 

Pig. 2 schematically shows an embodiment of the image signal interpolation 
unit according to the invention; 
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Fig 3 A schematically shows edges in two input images and in the interpolated 
image intermediate to these input images with a = 1/2 ; 

Kg 3B schematically shows edges in two input images and in the interpolated 
image intermediate to these input images with a m 1/4 • and 

5 Fig. 4 schematically shows an embodiment of the image processing apparatus 

according to the invention. 

Same reference numerals are used to denote similar parts throughout the figures. 



15 



10 m %- 1 schematically shows two input images 100 and 104 and an interpolated 

image 102 intermediate to these input images 100 and 104. The first input image 100 
corresponds to time n and the second input image 104 correspond to time n+1. The 
intermediate image 102 corresponds to time n+a, with 0 < or < 1. Hence, the first temporal 
distance between the first input image 100 and the interpolated image 102 is equal to ct and 
the second temporal distance between the second input image 1 04 and the interpolated image 
102 is equal tol-a. 

Aparttcularpixel 126 of the intermediate image 102 is computed on basis of 
an order statistical operation on a set of samples comprising a first group of samples 106-1 14 
which are derived from the first input image 100 and a second group of samples 1 16-124 

20 which are derived from the second input image 104. The relation between the first group of. 
samples 106-1 14 and the particular pixel 126 of the intermediate image 102 is determined by: 
the first motion vector Di . fit particular the first motion vector Di connects the particular 
pixel 126 and the central sample 110 of the first group of samples. The relation between the 
second group of samples 116-124 and the particular pixel 126 of the intermediate image 102 

25 is (ietermined by the second motion vector D2 . La particular the second motion vector D z 
connects the particular pixel 126 and the central sample 120 of the second group of samples. 

The first spatial distance ^between a first one 108 of the samples of the first 
group and a second one 106 of the samples of the first group is based on the first temporal 
distance* . The relation between die first spatial distance x t and the first temporal distance* 

30 is called the first quotient 

In the case that an analogue video signal is provided to the image signal 
interpolation unit according to the invention, then the acquisition of the samples of the first 
and second group can be directly controlled on basis of the first temporal distance a . 
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However, if the sampling of a received video signal, i.e. determining of pixel values, is 
perforated in another part of the image processing apparatus with a predetermined sampling 
grid, Le, fixed spatial distance, then alternative measures might be required to create the first 
group of samples 106-1 14. The first group of samples might be formed by taking only the 

S values of a part of the pixels in a particular neighborhood around the central pixel, which is 
connected to the first pixel of the interpolated image 102. That means that sub-sampling is 
applied, so e.g. each second or third pixel is taken. Alternatively the samples of the first 
group of samples 106-1 14 are computed by means of interpolation of available pixel values. 
The second spatial distance x» between a first one 118 of the samples of the 

10 second group and a second one 1 1 6 of the samples of the second group is based on the 

second temporal distancel - a > The relation between the seoond spatial distance x 2 and the 
second temporal distance t - a is called the second quotient. The first quotient is 
substantially equal to a second quotient: 

(l) 

1 5 Fig. 2 schematically shows an embodiment of the interpolation unit 200 

according to the invention. The interpolation unit 200 is provided at its input connector 210 
with a video signal comprising a first image 100 and a second image 104. The interpolation 
unit 200 is arranged to compute an interpolated image 102 intermediate the first 100 and the 
second image 104, The interpolated image 100 is located at a first predetermined temporal 

20 distance a fiom the first image 100 and is located at a second predetermined temporal 

distance 1 - a firom the second image 104. The interpolated image 102, the first image 100 
and the second image are provided at the output connector 212 of the interpolation unit 200. 
The interpolation unit 200 comprises; 

- a motion estimation unit 202 for furnishing a first Dt and a second Di motion vector 
25 relating to the first 100 and second image 102; 

- a first sample generation unit 204 for generating a first group of samples 106-1 14 on 

basis of values of pixels of the first image 100, the first motion vector Di and the first 
predetermined temporal distances , The required input is provided by means of the 
input connector 210, the motion estimation unit 202 and the control interface 214, 
30 respectively; 

- a second sample generation unit 206 for generating a second group of samples 116- 
124 on basis of values of pixels of the second image 104, the second motion vector 
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D 2 and the second predetermined temporal distance 1 - a . The required input is 
provided by means of the input connector 210, the motion estimation unit 202 and the 
control interfece 214, respectively; and 
- a central weighted median filter 208 for filtering of the samples of the first 106-U4 
5 and the second 1 16*-124 group to produce a first value of a first pixel of the 

interpolated image. 

The working of the interpolation unit 200 is as follows. Suppose that the value 
of a first pixel 126 of the interpolated image 102 has to be computed, with the first 
predetermined temporal distance a - l / 4 . The first motion vector A , related to the first 
10 pixel 126, is known. This first motion vector ~D\ links the first pixel 126 to a first particular 
pixel 1 10 in the first image 100. This first particular pixel 1 10 is taken as a central sample for 
the first group of samples. Subsequently, in the spatial environment of this first particular 
pixel, further samples are computed. Suppose the first image comprises a discrete set of 
pixels being disposed 1 mm fiom each other. Then the first group of samples comprises 
15 samples which have a first spatial distance x, fiom each other of V 4 mm. The first quotient is 
a 1/4 



20 



25 



* 1/4 



The first group of samples comprises 7 samples in total. 



30 



The second motion vector D 2 , related to the first pixel 126, is known. This 
second motion vector D 2 links the first pixel 126 to a second particular pixel 120 in the 
second image 120. This second particular pixel 120 is taken as a central sample for the 
second group of samples. Subsequently, in fee spatial environment of mis second particular 
pixel, further samples are computed. The second group of samples comprises samples which 
have a second spatial distance x 2 fiom each other of 3/4 mm. The second quotient is 
1 1 ec 3/4 

— — = - Tiie secciui group of samples comprises 7 samples in total. 

It will be clear that the first spatial distance jc, and the second spatial distance 
x z might also have been chosen to be both 4 times higher. That means that the first group of 
samples is based on a number of adjacent pixels and the second group of pixels is based on a 
number of pixels based on sub-sampling in the environment of the second particular pixel. 

The two groups of samples are provided to the central weighted median filter 
208. The central sample of the first group and the sample of the second group are weighted 
with relatively high weighting coefficients, i.e. equal to 7. The other samples with weighting 
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coefficients being equal to 1. That means that the weighting coefficients for the first group of 
samples are: 1,1,1,7,1,1 and 1, respectively and that the weighting coefficients fox the second 
group of samples are: 1,1,1,7,1,1 and 1, respectively. 

The eventual value of the first pixel 126 corresponds with the value of the 
5 median of the total set of samples. 

It will be clear that the groups of samples might also comprise more samples 
and that these samples might also be taken from a two-dimensional environment from the 
central pixel* 

The motion estimation unit 202 is e.g, as specified in the article "True-Motion 
10 Estimation with 3-D Recursive Search Block Matching" by G. de Haan et al. in IEEE 

Transactions on circuits and systems fbr video technology, voL3, no«5, October 1993, pages 
368-379. 

The generation of samples is defined by the following constraint: a first 
quotient is substantially equal to a second quotient, the first quotient being determined by a 

15 first spatial distance between a first one of the samples of the first group and a second one of 
the samples of the first group and the first predetermined temporal distance, the second 
quotient being determined by a second spatial distance between a first one of the samples of 
the second group and a second one of the samples of the second group and the second 
predetermined temporal distance* 

20 The motion estimation unit 202, the first sample generation unit 204 a the 

second sample generation unit 206 and the central weighted median filter 208 may be 
implemented using one processor. Normally, these functions axe performed under control of a 
software program product. During execution, normally the software program product is 
loaded into a memory, like a RAM, and executed from there, The program may be loaded 

25 from a background memory, like a ROM, hard disk, or magnetically and/or optical storage, 
or may be loaded via a network like Internet. Optionally an application specific integrated 
circuit provides the disclosed fiinctionality. 

Fig 3A schematically shows two input images 100 and 1 04 and an interpolated 
image 102 intermediate to these input images 100 and 104. The first input image 100 

30 corresponds to time n and the second input image 104 correspond to time n + 1 - The 

intermediate image 102 corresponds to time n + a , with a = 1/2 . Hence, the first temporal 
distance between the first input image 100 and the interpolated image 102 and the second 
temporal distance between the second input image 104 and the interpolated image 102 are 
mutually equal, in the first image 100 is a first edge 300, that means there is a transient in 
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15 



20 



25 



30 



luminance. Also in the second image 104 there is a second edge 304 corresponding to the 
first edge 300 in the first image. Because of lie appropriate selection of samples in the first 
image 100 and samples in the second image 104 the third edge 302 in the interpolated image 
is located at the cowect position, as can be seen by the mirror line 310 and the align line 3 12. 
The central sample 110 of the first image 100 has the largest weighting coefficient of all 
samples of the first group. The central sample 120 of the second image 104 has the largest 
weighting coefficient of all samples of the second group, 

Fig 3B schematically shows edges 300-304 in two input images 100, 104 and 
in the interpolated image 102 intermediate to these input images with a = 1/4. A difference 
with Fig. 3A is that the first spatial distance x, between samples derived from the first image 
100 is smaller man the second spatial distance x 2 between samples derived from the second 
image 104. 

Pig. 4 schematically shows an embodiment of the image processing apparatus 
400 according to the invention, comprising: 

- receiving means 402 for receiving a signal representing images; 

- the motion-compensated image signal interpolation unit 200 as described in 
connection with. Pig. 2 and 3; and 

- a display device 404 for displaying the output images of the motion-compensated 
image signal interpolation unit 200. This display device 406 is optional. 

The signal may be a broadcast signal received via an antenna at cable but may also be a 
signal from a storage device like a VCR (Video Cassette Recorder) or Digital Versatile Disk 
(DVD). The signal is provided at the input connector 41 0. The image processing apparatus 
400 might e.g. be a TV. Alternatively the image processing apparatus 400 does not comprise 
the optional display device but provides HD images to an apparatus that does comprise a 
display device 406. Then the image processing apparatus 400 might be e.g. a set top box, a 
sateffite-tuner, a VCR player or a DVD player. But it might also be a system being applied by 
a film-studio or broadcaster. 

It should be noted that the above-mentioned embodiments illustrate rather than 
limit the invention and that those skilled in the art will be able to design alternative 
embodiments without departing from the scope of the appended claims, hi the claims, any 
reference signs placed between parentheses shall not be constructed as limiting the claim. 
The word 'comprising* does not exclude the presence of elements or steps not listed in a 
claim. The word "a" or "an" preceding an element does not exclude the presence of a 
plurality of such elements. The invention can be implemented by means of hardware 
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comprising several distinct elements and by means of a suitable programmed computer. In 
the unit claims enumerating several means, several of these means can be embodied by one 
and the same item of hardware, 
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CLAIMS; 



1 . A motion-compensated image signal interpolation unit (200) for generating an 
interpolated image intermediate a first and a second image, the interpolated image 

5 (102) being located at a first predetermined temporal distance (a ) from the first 

image and being located at a second predetermined temporal distance (1 - « ) from 
the second image, the interpolation unit (200) comprising: 
" motion estimation means (202) for fttinishmg a first and a second motion vector 
relating to the first and second image; 
10 - fiiniishing means (204, 206) for furnishing a first group of samples on basis of 

values of pixels of the first image and the first motion vector and for furnishing a 
second group of samples on basis of values of pixels of the second image and the 
second motion vector; and 

- filtering means (212) for ordered statistical filtering of the samples of the first and 
15 me second group to produce a first value of a first pixel of the interpolated image 

(102), whereby a first quotient is substantially equal to a second quotient, the first 
quotient being determined by a first spatial distance ( x,) between a first one of the 
samples of the first group and a second one of the samples of the first group and the 
first predetermined temporal distance ( a ), the second quotient being determined by a 
20 second spatial distance ( x % ) between a first one of the samples of the second group : 

and a second one of the samples of the second group and the second predetermined 
temporal distance (1 - a ). 

2. A motion-compensated image signal interpolation unit (200) as claimed in Claim 1, 
25 whereby the first spatial distance is proportional to the first predetermined temporal 

distance! 

3. A motion-compensated image signal interpolation unit (200) as claimed in Claim 1, 
whereby the filtering means (212) include a median filter. 

30 

4. A motion-compensated image signal interpolation unit (200) as claimed in Claim 3, 
whereby the filtering means (212) include a weighted median filter. 



5. A motion-compensated image signal interpolation unit (200) as claimed in Claim 4, 
35 whereby a particular weighting coefficient of the weighted median filter for weighting 

a particular sample of the first group of samples is higher than each of the further 
weighting coefficients for weighting further respective samples of the first group of 
samples, the particular sample bang located in the center of the first group of 
samples. 
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6. A motion-compensated image signal interpolation unit (200) as claimed in Claim 5, 
Whereby the particular weighting coefficient is higher than a sum of die further 
weighting coefficients. 

5 

7. A motion-compensated image signal interpolation unit (200) as claimed in Claim I, 
whereby a second value corresponding to the first one of the samples of the first 
group equate a third value of a third one of the pixels of the first image. 

10 8. A motion-compensated image signal interpolation unit (200) as claimed in Claim 1, 
whereby a second value corresponding to the first one of the samples of the first 
group is computed by means of interpolation of a third value of a third one of the 
pixels of the first image and a fourth value of a fourth one of the pixels of the first 
image in a spatial environment of the third one of the pixels. 

15 

9, An image processing apparatus (400) comprising: 

- receiving means (402) for receiving an image signal representing a first and a 
second image; and 

- a motion-compensated image signal interpolation unit (200) coupled to the receiving 
20 means, for generating an interpolated image (102) intermediate the first and the 

second image, the interpolated image (102) being located at a first predetermined 
temporal distance from the first image and being located at a second predetermined 
temporal distance from the second image, the interpolation unit (200) as claimed in 
Claim 1. 



25 



10. An image processing apparatus as claimed in Claim 9, further comprising a display 
device fbr displaying the interpolated image (102). 



11. A method of generating an interpolated image (102) intermediate a first and a second 
30 image, the interpolated image (102) being located at a first predetermined temporal 

distance from the first image and being located at a second predetermined temporal 
distance from the second image, the method comprising: 

- fbtnishing a first and a second motion vector relating to the first and second image; 

- furnishing a first group of samples on basis of values of pixels of the first image and 
35 the first motion vector and for furnishing a second group of samples on basis of 

values of pixels of the second image and the second motion vector; and 

- ordered statistical filtering of the samples of the first and the second group to 
produce a first value of a first pixel of the interpolated image (102), whereby a spatial 
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distance between a first one of the samples of the first group and a second one of the 
samples of the first group is based on the predetermined temporal distance. 

12. A computer program product to he loaded by a computer arrangement, comprising 
5 instructions to generate an interpolated image (102) Intermediate a first and a second 

image, the interpolated image (102) being located at a first predetermined temporal 
distance from the first image and being located at a second predetermined temporal 
distance from the second image, the computer arrangement comprising processing 
means and a memory, the computer program product after being loaded, providing 
10 said processing means with the capability to carry out: 

- furnishing a first and a second motion vector relating to the first and second image; 

- furnishing a first group of samples on basis of values of pixels of the first image and 
the first motion vector and for furnishing a second group of samples on basis of 
values of pixels of the second image and the second motion vector; and 

15 - ordered statistical filtering of the samples of the first and the second group to 

produce a first value of a first pixel of the interpolated image (102), whereby a spatial 
distance between a first one of the samples of the first group and a second one of the 
samples of the first group is based on the predetermined temporal distance 
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ABSTRACT: 



A motion-compensated image signal interpolation unit (200) for generating an interpolated 
5 image being located at a first predetermined temporal distance ( a ) ftom the first image and 
being located at a second predetermined temporal distance (1 - a ) ftom the second image is 
disclosed. The interpolation unit (200) comprises: motion estimation means (202) for 
furnishing a first and a second motion vector relating to the first and second image; 
furnishing means (204, 206) for furnishing a first group of samples on basis of values of 

10 pixels of the first image and the first motion vector and for furnishing a second group of 
samples on basis of values of pixels of the second image and the second motion vector; and 
filtering means (212) for ordered statistical filtering of the samples of the first and the second 
group to produce a first value of a first pixel of the interpolated image (1 02), whereby a first 
quotient is substantially equal to a second quotient, the first quotient being determined by a 

15 first spatial distance ( x x ) between a first one of the samples of the first group and a second 
one of the samples of the first group and the first predetermined temporal distance {a ), the 
second quotient being determined by a second spatial distance (x 2 ) between a first one of the 
samples of the second group and a second one of the samples of the second group and the 
second predetermined temporal distance (1 - a ). 

20 



Fig.l 
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